<html>
<head><meta charset="utf-8"><title>Parallel iteration with smol · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html">Parallel iteration with smol</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="211397510"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211397510" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211397510">(Sep 27 2020 at 04:16)</a>:</h4>
<p>I'm in an async context using smol, and I'm trying to figure out the best way to run a bunch of things in parallel for performance. I have an <code>Mmap</code> of a file, and I'd like to process sections of it in parallel, but I also need to collect the output of that parallel processing in order. Also, each of the individual steps produces a <code>Result</code> that might be an <code>Err</code>.</p>



<a name="211397515"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211397515" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211397515">(Sep 27 2020 at 04:16)</a>:</h4>
<p>Any suggestions for a good starting point?</p>



<a name="211397516"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211397516" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211397516">(Sep 27 2020 at 04:16)</a>:</h4>
<p>cc <span class="user-mention" data-user-id="128448">@stjepang</span></p>



<a name="211398427"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211398427" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211398427">(Sep 27 2020 at 04:46)</a>:</h4>
<p>I feel like there's something I'm completely missing. I've used rayon before, which was extremely straightforward.</p>



<a name="211420616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211420616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Philipp Korber <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211420616">(Sep 27 2020 at 15:14)</a>:</h4>
<p>I think the main question is weather or not the parallel processing is blocking or non-blocking and in turn weather or not you want to<br>
"fork out" the work into a different thread pool then the one done for the async I/O.</p>
<p>This might also change weather or not you have need to poll the futures for the resulting chunks in parallel or not.</p>
<p><code>join_all</code> lets you poll futures in parallel and wait for all to be completed.</p>
<p><code>FuturesOrdered</code>/<code>StreamExt::buffered(number_of_chunks)</code> gives you a stream which polls the futures in parallel and returns the resulting items in order (in your case this would be a stream of <code>Results</code>.</p>
<p>I don't know about Mmap and if liftimes get in your way. </p>
<p>Generally async if focused on non-blocking I/O and rayon on parallel computation, I don't know if there is a "rayon for async" (<strong>anyone knows of such a crate?</strong>).</p>



<a name="211420651"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211420651" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Philipp Korber <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211420651">(Sep 27 2020 at 15:15)</a>:</h4>
<p>Also note that while smol by default used <code>futures_lite</code> all the <code>std::future</code> based futures libraries can be combined freely without problems. The buffered method comes from the <code>futures</code> crate.</p>



<a name="211421100"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211421100" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211421100">(Sep 27 2020 at 15:26)</a>:</h4>
<p>Parallel or concurrent? The latter can happen with <code>FuturesOrdered</code> as mentioned above. For the former you need to either tie your code to the runtime and spawn tasks (and then later gather them, in order, which you can do with <code>FuturesOrdered</code> as well) or use an external thread pool (to much the same effect).</p>



<a name="211421218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211421218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211421218">(Sep 27 2020 at 15:29)</a>:</h4>
<p>the caveat with using tasks for parallelisation of blocking tasks is that you can accidentally exhaust the executor's thread pool and end up blocking the non-blocking code from progressing any further until the blocking code is done.</p>



<a name="211422490"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211422490" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211422490">(Sep 27 2020 at 16:01)</a>:</h4>
<p>I still want to get some kind of <code>spawn_future</code> into rayon:<br>
<a href="https://github.com/rayon-rs/rayon/pull/679">https://github.com/rayon-rs/rayon/pull/679</a></p>



<a name="211422565"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211422565" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211422565">(Sep 27 2020 at 16:02)</a>:</h4>
<p>The debate was whether my simple map from <code>FnOnce</code> to <code>Future</code> is enough, or if we should really map <code>Future</code> to <code>Future</code></p>



<a name="211422649"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211422649" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211422649">(Sep 27 2020 at 16:04)</a>:</h4>
<p>I don't really want to teach rayon to be an executor though, and I think users can just as well <code>.await</code> the input in their own context before calling rayon</p>



<a name="211422975"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211422975" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211422975">(Sep 27 2020 at 16:12)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> maybe you can just use a one-shot futures channel? Spawn a rayon job with the sender, then <code>.await</code> the receiver in your async context</p>



<a name="211423230"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211423230" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211423230">(Sep 27 2020 at 16:19)</a>:</h4>
<p>The problem is that I need to do an <code>.await</code> <em>inside</em> each parallel job.</p>



<a name="211423240"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211423240" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211423240">(Sep 27 2020 at 16:19)</a>:</h4>
<p>(Specifically, a rusoto call.)</p>



<a name="211423278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211423278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211423278">(Sep 27 2020 at 16:20)</a>:</h4>
<p>Ah, I have no rayon solution for that</p>



<a name="211423299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211423299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211423299">(Sep 27 2020 at 16:20)</a>:</h4>
<p>(unless we go full executor)</p>



<a name="211423382"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211423382" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211423382">(Sep 27 2020 at 16:23)</a>:</h4>
<p>To answer an earlier question, I would be happy to share the executor's normal thread pool; that's what I would hope for.</p>



<a name="211423604"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211423604" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Philipp Korber <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211423604">(Sep 27 2020 at 16:28)</a>:</h4>
<p>Do you need non 'static lifetimes/borrowed from the chunk processing to the outside?</p>



<a name="211423690"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211423690" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Philipp Korber <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211423690">(Sep 27 2020 at 16:31)</a>:</h4>
<p>(if so this would be a bit tricky)</p>



<a name="211423911"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211423911" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211423911">(Sep 27 2020 at 16:37)</a>:</h4>
<p>Not this time, no. But I do ideally need non-<code>'static</code> lifetimes borrowed from the code just above the parallel processing <em>into</em> the chunks.</p>



<a name="211424022"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211424022" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211424022">(Sep 27 2020 at 16:40)</a>:</h4>
<p>I need to do parallel processing on chunks of a buffer.</p>



<a name="211424031"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211424031" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211424031">(Sep 27 2020 at 16:40)</a>:</h4>
<p>Or more generally, chunks of a file.</p>



<a name="211424109"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211424109" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211424109">(Sep 27 2020 at 16:42)</a>:</h4>
<p>For each parallel chunk, I need to do a small amount of CPU-based computation (a hash), then make a request that will require a <code>.await</code>, then yield a value. And I need to process the yielded values in order, either concurrently or afterwards.</p>



<a name="211424126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211424126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211424126">(Sep 27 2020 at 16:43)</a>:</h4>
<p>(technically, the yielded value does not depend on the request; it's just the value of that hash)</p>



<a name="211438989"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211438989" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Philipp Korber <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211438989">(Sep 27 2020 at 23:05)</a>:</h4>
<p>You should be able to do one of the two  approached shown in this example:</p>
<p><a href="https://play.rust-lang.org/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=915937a1c50d6b2e8eed428aa51e8262">https://play.rust-lang.org/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=915937a1c50d6b2e8eed428aa51e8262</a></p>
<p>WARNING: This won't run on <a href="http://play.rust-lang.org">play.rust-lang.org</a> as you can't use smol there! (Also that's just fastely written down, so style isn't perfect and I forgot to run <code>cargo fmt</code> <span aria-label="stuck out tongue wink" class="emoji emoji-1f61c" role="img" title="stuck out tongue wink">:stuck_out_tongue_wink:</span> )</p>



<a name="211457645"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211457645" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211457645">(Sep 28 2020 at 07:00)</a>:</h4>
<p>That looks really promising! Will investigate tomorrow.</p>



<a name="211472343"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211472343" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Philipp Korber <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211472343">(Sep 28 2020 at 09:58)</a>:</h4>
<p>Instead of <code>FuturesOrdered</code> you can also use <code>smol::LocalExecutor</code> but it's a bit more complex/roundabout.</p>



<a name="211501012"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Parallel%20iteration%20with%20smol/near/211501012" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Parallel.20iteration.20with.20smol.html#211501012">(Sep 28 2020 at 14:39)</a>:</h4>
<p>(small nit: you can push directly into <code>FuturesOrdered</code> without the intermediate vector)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>